gh-109559: Update unicodedata for Unicode 15.1
#109560
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
SnoopJ
commented
Sep 19, 2023
| ('2B740', '2B81D'), # CJK Ideograph Extension D | ||
| ('2B820', '2CEA1'), # CJK Ideograph Extension E | ||
| ('2CEB0', '2EBE0'), # CJK Ideograph Extension F | ||
| ('2EBF0', '2EE5D'), # CJK Ideograph Extension I |
There was a problem hiding this comment.
The range check that occurs later in this file implicitly assumes this list is in sorted order. It seems simpler to have an idiosyncratic order here than to try to introduce sorted() or somesuch.
SnoopJ
commented
Sep 19, 2023
Comment on lines
+1109
to
+1112
| for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded(): | ||
| if propinfo: | ||
| # this is not a binary property, ignore it | ||
| continue |
There was a problem hiding this comment.
All the properties defined in DerivedCoreProperties.txt happened to be binary until the latest update, so this tool was getting away with the assumption that they are always binary.
As of Unicode 15.1, this file also includes definitions that use the Indict_Conjunct_Break (InCB) property, which is an enumeration.
With this change, the loop skips over any non-binary properties, since we have nothing to do with them.
There was a problem hiding this comment.
It seems like it would be safer to explicitly ignore Indict_Conjunct_Break rather than throw out everything with a second column.
There was a problem hiding this comment.
Is there a particular failure mode you have in mind? My rationale here was that the current internalized DB only cares about binary properties in this file, but in practice any of the property types enumerated by UAX#44 could appear in a future revision.
I'm not strongly opposed to ignoring the specific property that breaks the tool against the current revision, but my rationale was that it seems safer to prevent this class of failure in the future if/when additional non-binary properties are added.
benjaminp
reviewed
Sep 19, 2023
Comment on lines
+1109
to
+1112
| for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded(): | ||
| if propinfo: | ||
| # this is not a binary property, ignore it | ||
| continue |
There was a problem hiding this comment.
It seems like it would be safer to explicitly ignore Indict_Conjunct_Break rather than throw out everything with a second column.
Good catch! The only reference I could find to Unicode 15.0.0 was in |
That's the one you shouldn't touch because it's generated. :) See references in |
|
Whoops! Guess my initial grep was poorly-formed. Thanks for the guidance. |
|
All commit authors signed the Contributor License Agreement. |
benjaminp
force-pushed
the
feature/Unicode-15.1
branch
from
ca72e1c to
44f6770
Compare
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Whoops, sorry about the breakage, I forgot to include |
csm10495
pushed a commit
to csm10495/cpython
that referenced
this pull request
Sep 28, 2023
…nGH-109560) --------- Co-authored-by: Benjamin Peterson <[email protected]>
Glyphack
pushed a commit
to Glyphack/cpython
that referenced
this pull request
Sep 2, 2024
…nGH-109560) --------- Co-authored-by: Benjamin Peterson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changeset implements #109559, adding Unicode 15.1 support to the internal databases that support the
unicodedatamodule. The bulk of this Unicode update is the addition of a new CJK Ideograph Extension, which requires updating the CJK ranges baked into themakeunicodedata.pytool andunicodedata.c.I have also taken the liberty of making the documentation of CJK Ideograph Extension ranges consistent between
makeunicodedata.pyandunicodedata.c(i.e. each range includes a comment naming the range).